Limit MongoDB memory use on Windows without Virtualization

I’ve seen the question of how to control MongoDB’s memory usage on Windows come up several times and the stock answer always seemed to be “you can’t – it uses memory-mapped files and if you want to limit resources you need to use some form of virtualization to do it (HyperV, VMWare, Virtuozzo etc…)”.

win2008r2

If you are using MongoDB on a dedicated server then you generally want it to use all the memory it can but if you want to use it on a server shared with other processes (e.g. an IIS website using MongoDB for storage, maybe with SQL Server as well) then you will want to put a cap on how much it uses to ensure memory is kept available for the other processes.

So is it possible if you are not on a virtualized environment? Yes (otherwise this would be a very short blog post!) and we’ll explore how …

The standard behaviour described above is actually a result of the default resource manager used by Windows but both Windows 2003 and Windows 2008 have a separate installable option called the “Windows System Resource Manager” (WSRM) that allows greater control over the CPU and Memory available to a process.

First of all, lets look at what we’re trying to solve. Here we have a low-memory server (only 2Gb) running MongoDB on Windows 2008 R2 x64. There are a few databases of a few Gb each so the mongod.exe process quickly starts consuming as much memory as it can (rightly so) to keep as much of it’s indexes in memory for the fast performance we know and love:

2-performance

1-processes

What we’d like to do is save some memory for other processes by limiting the mongod.exe process to 1Gb in this case (I know this is ridiculously low but the only thing that will change for you are the actual limits you want to use).

To do this we first need to install Windows System Resource Manager which on Windows 2008 is available under the Features section of the Server Manager.

windows-features

Once that’s installed fire it up and you’ll see the default resource management policies. By default the standard Windows “memory is given to whoever shouts loudest” policy is used but other pre-configured alternatives are available. WSRM also provides a calendar / event system where the policy can be changed at certain times (a typical scenario is giving critical business apps priority during the day but then batch processes greater priority overnight). We’re not going to go into the calendar features here but it’s interesting to know about.

Let’s create a new policy to control the resources that MongoDB can consume. To do this, right click on the “Resource Allocation Policies” container and chose “New Resource Allocation Policy …”. This will present us with the New Resource Allocation Policy dialog below:

6-new

First of all, we need to add a new resource allocation entry so click the ‘Add…’ button and we get to another “Add or Edit Resource Allocation” dialog:

7-add

We don’t have a Process matching criteria for MongoDB yet so choose <New…> to get … yes, you guessed – another dialog, this time “New Process Matching Criteria”. We’ll call it “mongod_process” and click the Add… button to get another death-by-dialog to define it.

There are a few ways to do this – if MongoDB is installed as a service then you can choose “Registered Service” in the drop-down, click “Select” and choose it from the list or you can select from a list of running processes or you can just enter the full path and filename to mongod.exe. Here is the entry after selecting an installed MongoDB Windows Service:

9-edit-rule

After clicking OK we get back to the Process Matching Criteria dialog showing our new rule:

10-mongod-process-criteria

After clicking OK we’re now back at the Resource Allocation dialog with the new “mongod_process” Process matching criteria selected and can now decide what resources we want to allocate to the process. Lets limit the CPU to 50% (not that MongoDB seems to consume much CPU):

11-allocation-general

The Memory tab allows us to limit the memory and here there are two options. The maximum committed memory limit is more to control apps that may have a memory leak and can be setup to stop or alert someone when the process goes above the configured limit. We don’t want this one … instead we’ll set a maximum working set limit which will control how much memory is allocated to MongoDB. In this case, we’ll set the limit to 1Gb but the actual value to use will depend on your circumstances:

12-allocation-memory

After clicking OK we should then be at the Resource Allocation Policy dialog with our process matching criteria, CPU and memory limits shown. We could include more limits in the policy but we’ll leave it as it is for now – any remaining resources will be allocated to other processes as normal after the limits have been imposed.

13-limit-mongo-policy

The final piece is to make this policy active which is done by clicking on the “Selected Policy” link on the main ‘page’ or right-clicking on the new entry under the “Resource Allocation Policies” and choosing “Set as Managing Policy’”. You can also right-click on the “Windows System Resource Manager (local)” entry and choose “Properties …” to display the dialog below which allows you to select the Current resource allocation policy:

14-properties

So, we’ve created a new policy that has a criteria to match the mongod.exe process which will limit the CPU usage to 50% and memory to 1Gb … does it work? Here’s the result after it’s enabled showing the memory used immediately dropping:

15-performance

… and the MongoDB / mongod.exe process using the 1Gb limit we specified (1Gb = 1024Mb = 1,048,576Kb).

16-processes

So, we’ve successfully limited the CPU and memory that MongoDB can consume without havign to resort to any form of server-virtualization and while MongoDB will probably not run as fast as it did when it had free-reign to consume as much as it wanted (or rather, when the default windows resource manager gave it what it asked for) we will probably have a faster overall system as our other processes are allocated the memory and CPU that they need for a better balanced system.

Please let me know what you think of the above technique and if you find it useful.

About these ads

19 thoughts on “Limit MongoDB memory use on Windows without Virtualization

  1. Great post Simon! I’ve been wondering if this was possible for a while, but never got around to trying it out. It’s really useful to now have the option of running MongoDB on the same machine as IIS and other apps, instead of needing to give it an entire box or VM to itself.

  2. Pingback: ehcache.net

  3. Life saver! Due to resource constraints we’ve got MongoDB running alongside SQL Server and a few other things, and MongoDB eats all the pies even if it’s not hungry and there’s no pie left for anyone else. It’s DB gluttony.

    Mmmm pies.

  4. Hi! Thanks for great tutorial.
    I faced an interesting situation: when I’m setting limit to 1gb it works fine, but when I’m setting 4 or 8gb it doesn’t work. mongo eating all free ram(15.2 from 16gb).
    Any ideas what I’m doing wrong?

    • I have the same issue. Setting it to sup 1GB it all works fine. Any number above it and the policy wont kick in.
      Not sure if it’s a bug in Resource Manager, but it’s not working.

  5. It’s not clear to me this is necessary. The memory usage by mongo is primary just space in the file system cache. The OS should yield that space to other processes as they need it.

    • If you’ve ever used Mongo with 10GB< data on Windows you'd know that Mongo's RAM strategy just isn't working.
      Once Mongo's "borrowed" all the memory the machine will be extremely slow.

  6. Is it possible to install Windows System Resource Manager on Windows 7? I am doing some R&D work, and don’t have Windows 2008 handy.

  7. Pingback: Limit Mongo CPU and Memory usage on Windows 2008 R2 | Avivo Tech Blog

  8. This works until I specify more than 2GB of memory to mongo, then it quits working. 2GB works, 3GB does not work, it will grow and use all memory. Very strange, Trying a couple of values in between to see where it breaks. Any ideas?

    • Now see post just above mine regarding this issue. Unfortunately, the connect.microsoft.com link is no longer valid.

  9. Pingback: Windows System Resource Manager working set memory limit not working above 2GB | BlogoSfera

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s