Seq Documentation and Support

Seq Documentation and Support

Welcome to the Seq documentation hub. You'll find comprehensive guides and documentation to help you start working with Seq as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    
Ask A Question

Questions

1

Unexpected 'Memory is overutilized' and stopping then blocking SEQ

Hi SEQ Team, This morning we had a problem. From log can be seen as below. {"@t":"2017-12-25T01:00:20.8747699Z","@mt":" Seq stopped cleanly"} {"@t":"2017-12-25T01:00:05.3491070Z","@mt":" Stopping {AppCount} running apps","AppCount":13} {"@t":"2017-12-25T01:00:05.0416443Z","@mt":" Stopping broadcast channel","SourceContext":"Flare.Events.Broadcast.BroadcastChannel"} {"@t":"2017-12-25T01:00:05.0416443Z","@mt":" Stopping {Tasks} scheduled tasks","Tasks":11,"SourceContext":"Seq.Server.Tasks.TaskRunner"} {"@t":"2017-12-25T01:00:03.7126398Z","@mt":" Signalling for {Workers} background worker(s) to complete","Workers":3,"SourceContext":"Flare.Workers.WorkerPool"} {"@t":"2017-12-25T01:00:03.7126398Z","@mt":" Stopping HTTP listener"} {"@t":"2017-12-25T00:58:39.3548640Z","@mt":" Memory is overutilized, so cooling down segment {Segment}","Segment":"12/23/2017 12:00:00","SourceContext":"Flare.Events.Cache.SegmentCache"} {"@t":"2017-12-25T00:58:30.5771826Z","@mt":" Memory is overutilized, so cooling down segment {Segment}","Segment":"12/23/2017 09:00:00","SourceContext":"Flare.Events.Cache.SegmentCache"} {"@t":"2017-12-25T00:58:21.4577578Z","@mt":" Memory is overutilized, so cooling down segment {Segment}","Segment":"12/23/2017 06:00:00","SourceContext":"Flare.Events.Cache.SegmentCache"} {"@t":"2017-12-25T00:58:12.0509496Z","@mt":" Memory is overutilized, so cooling down segment {Segment}","Segment":"12/23/2017 03:00:00","SourceContext":"Flare.Events.Cache.SegmentCache"} Then our PSScheduledJob that is checking if SEQ is in running state (seq status) discovered that is not in running state. At 1:30AM tried to start the SEQ (SEQ start). But: {"@t":"2017-12-25T01:30:13.5089970Z","@mt":" Uncaught exception from worker {WorkDescription} ({WorkId})","@l":"Error","@x":"Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = AppRunner (ReflectionActivator), Services = [Seq.Server.Features.Runner.IAppRunner, Flare.Diagnostics.Reporting.IDiagnosticReporter], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = QuinceDB (DelegateActivator), Services = [Quince.QuinceDB], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = StorageSubsystem (DelegateActivator), Services = [Seq.Server.Storage.StorageSubsystem], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Cannot access file, the file is locked or in use (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = QuinceDB (DelegateActivator), Services = [Quince.QuinceDB], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = StorageSubsystem (DelegateActivator), Services = [Seq.Server.Storage.StorageSubsystem], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Cannot access file, the file is locked or in use (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = StorageSubsystem (DelegateActivator), Services = [Seq.Server.Storage.StorageSubsystem], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Cannot access file, the file is locked or in use (See inner exception for details.) ---> Microsoft.Isam.Esent.Interop.EsentFileAccessDeniedException: Cannot access file, the file is locked or in use\r\n at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err)\r\n at Flare.Storage.Esent.Data.EsentDBSession..ctor(Instance instance, String path, Action notifyDisposed)\r\n at Flare.Storage.Esent.Data.EsentDB.BeginSession(Action notifyDisposed, String tag)\r\n at Flare.Storage.Esent.Migration.EventStoreMigrator.GetAppliedMigrations(EsentDB db)\r\n at Flare.Storage.Esent.Migration.EventStoreMigrator.Migrate(EsentDB db)\r\n at Flare.Storage.Esent.EsentStorageExtent.BeginSession(String tag)\r\n at Flare.Storage.Esent.EsentStorageExtent.<Get>d__15.MoveNext()\r\n at Flare.Events.EventStore.<Get>d__12.MoveNext()\r\n at System.Linq.Enumerable.<TakeIterator>d__24`1.MoveNext()\r\n at Flare.Events.EventStore..ctor(IStorageEngine[] storageEngines, String extentsPath, WorkerPool workerPool, MemoryThresholds memoryThresholds, BroadcastChannel broadcast)\r\n at Seq.Server.SeqServerModule.InitStorageSubsystem(IComponentContext c)\r\n at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\r\n at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\r\n at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n --- End of inner exception stack trace ---\r\n at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)\r\n at Autofac.Core.Resolving.InstanceLookup.Execute()\r\n at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)\r\n at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\r\n at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)\r\n at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable`1 parameters)\r\n at Seq.Server.SeqServerModule.<>c.<Load>b__16_7(IComponentContext c)\r\n at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\r\n at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\r\n at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n --- End of inner exception stack trace ---\r\n at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)\r\n at Autofac.Core.Resolving.InstanceLookup.Execute()\r\n at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)\r\n at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()\r\n at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\r\n at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n --- End of inner exception stack trace ---\r\n at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)\r\n at Autofac.Core.Resolving.InstanceLookup.Execute()\r\n at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)\r\n at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)\r\n at Autofac.Features.LazyDependencies.LazyRegistrationSource.<>c__DisplayClass5_0`1.<CreateLazyRegistration>b__1()\r\n at System.Lazy`1.CreateValue()\r\n at System.Lazy`1.LazyInitValue()\r\n at Seq.Server.ServiceProcess.ServerService.<Start>b__9_0(CancellationToken cancel)\r\n at Flare.Workers.WorkerPool.<>c__DisplayClass4_0.<Run>b__0()","WorkDescription":"Start background processes","WorkId":"908e9827-90dd-446b-b9ef-cd5d09fd2b79","SourceContext":"Flare.Workers.WorkerPool"} {"@t":"2017-12-25T01:30:03.3546165Z","@mt":" Available storage engines in order of preference are {Engines}","Engines":["ESENT","LMDB"],"SourceContext":"Flare.Events.EventStore"} {"@t":"2017-12-25T01:30:03.3546165Z","@mt":" Opening event store at {ExtentsPath}","ExtentsPath":"F:\\SEQ\\Extents","SourceContext":"Flare.Events.EventStore"} {"@t":"2017-12-25T01:30:03.3233496Z","@mt":" ESENT sparse file support is available on this system"} {"@t":"2017-12-25T01:30:03.0889876Z","@mt":" Opening document store {DatabasePath}","DatabasePath":"F:\\SEQ\\Documents\\default.quince","InstanceName":"Quince","SourceContext":"Quince.QuinceDB"} {"@t":"2017-12-25T01:30:03.0577566Z","@mt":" Seq listening on {ListenUris}","ListenUris":["https://seq.somename.com:5341/"]} {"@t":"2017-12-25T01:30:02.4483350Z","@mt":" Seq {SeqVersion} {ProcessWordSize}-bit running on .NET {DotNetVersion} and OS {OSCaption} ({OSVersion}) {OSWordSize}-bit","SeqVersion":"4.1.17","ProcessWordSize":64,"DotNetVersion":"4.0.30319.42000","OSCaption":"Microsoft Windows Server 2012 R2 Datacenter","OSVersion":"6.3.9600","OSWordSize":64,"SourceContext":"Seq.Server.Features.Diagnostics.EnvironmentInformationWriter"} Then during working hours when manually checking SEQ status surprisingly: The Seq service is installed and running. SEQ.exe process allocated only 20 MB of RAM when normally 21GB. Trying to reach SEQ endpoint then request returned: Error Seq in unavailable. Failed to initialize storage: Cannot access file, the file is locked or in use. The whole problem showed up when no additional workload or maintenance has been performed. The problem has been solved by manual Stop-Process SEQ -Force then SEQ start. Looking forward to a suggestion.

Posted by Igor about a month ago