No segments file found in ramdirectory

Hi Uwe, I was curious what this means for the solr documentCache? If this is all essentially loaded into memory upon retrieval is there much point in needing it -- other than perhaps desiring autowarming? The other caches seem more relevant for continued use, but documentCache seems redundant. Just like Lucene TopDocs, which has no cache at all and is very fast in displaying something like 20 documents, I would disable this cache.

I generally recommend that to my customers for Lucene code to not cache Document objects, unless they do some result-post-processing fetching thousands of search result document objects.

This is no longer the case since 2. Hi Uwe, Thanks for this interesting article. This also improves the garbage collector so it distributes the allocated objects correctly to CPU local memory. But in practise, we have not seen MySQL-like problems. The reason for this is, that the maximum size of memory, Java can allocate for a single object is 2 GiB new byte[Integer. Also the JDK will use libnuma, to distribute the allocated blocks accoring to their requirements.

Hi Uwe, Like you have mentioned that it's best to disable documentCache. Wouldn't these get effected too if we reduce heap space? Also during indexing if the heap space is low could it give me heap space errors during a merge? FieldCache is needed for sorting and cannot be turned off, but may get obsolete with the introduction of Lucene QueryResultCache is like FilterCache caching results, but for queries it also caches the score values, so a simple bitset is not enough.

If you have very expensive queries that repeat quite often like Dismax , you should use it. Hi Uwe, Thanks for explaining the various caches.

Indeed once DocValues get integrated into the schema using them for sorting and faceting. One this I'm still not clear about it how to balance between allocating heap space for running search.

I know you can't really put a formula to it but as far as my understanding goes, Lucene benefits from having more free memory with the os but Solr being an application benefits from having more heap space. Why I'm so keen on this balance is I want to make sure I don't run out of memory when committing the index or if I have a query with multiple facets and sort on fields while maintaining good search performance. Hi Uwe, I'm linux support for some developers trying to implement lucene and make use of mmapdirectory.

I'm trying to figure out exactly what I need to do to facilitate this for them. Are there any kernel tweaks or anything that I need to do on the system? It's a RHEL 5. If you could point me to any documentation from a systems, rather than a developers perspective, it would be greatly appreciated. Vielen Dank, Jay Herbig. Hi Jay, unfortunately, I am also on the developer perspective But I can answer parts of your questions, as I am also involved in managing Lucene-based servers: - There are no special kernel-settings needed, the default kernels in all Linux distributions should support mmap, as this is required by POSIX and the ld.

The problem are as noted before some restrictive settings with ulimit: - Lucene in general needs a large number of open files, so you should in all cases raise the maximum open file count per process. On servers only running lucene apps and nothing more which we recommend , the upper limit is just useless and should be raised to some maximum like open files depends on kernel.

If you don't want to raise it too high, use at a minimum - this is not really specific to mmap, applies all Lucene storage implementions.

As my article says, this has nothing to do with physical RAM, its only the size of address space the app can occupy. The setting of virtual mem is only important on 32 bit systems, where one process could easily allocate the whole virtual 32 bit off address space, bringing the server down. On 64 bit this is unlikely to happen :- Max memory size "ulimit -m" should also be unlimited.

Both settings are unlimited in most linux distribs like Ubuntu, Debian. There may be other settings in ulimit like maximum file size, but I don't think you would have any limits by default.

In any case, if I get link to an article about "sys admin's settings", I will post it here. Thanks Uwe, for sharing such valuable information. Do you have any recommendation on the of CPUs too? Hi Uwe, thanks for sharing thoughts on mmap impl for linux 64 bits.

Is there any additional recommendation when using different hosts of appservers sharing indexes stored in NFS? Only one appserver instance writes to index, the other instances only read. Great article! And if I do, what is the preferred Directory to use? Only code for writing is missing in Lucene. If I have time, I will work on this. The problems with writing and also RAMDirectory in general are the fact that in older Lucene versions, we don't really know how big the files are.

So Garbage collector has to handle millions of small byte[] if you have a large index in RAMDirectory. In Lucene 4. Otherwise, if it is on disk, MMapDirectory is the way to go. While your article is helpful in some aspects, it is trivially simplistic in others. Nothing is for "free". Regardless of how much virtual address space is available, the physical address space is limited.

Your many points about the inefficiency of the java heap are generally correct. Yes, it's true that memory-mapped files are more efficient for loading an index into physical memory. However, in other ways your praise of memory mapping the virtual address space is trivial and simplistic.

If the index is so large that there are very large amounts of virtual memory from the memory mapped files being loaded into physical memory, then this crowds out the physical memory in use by the app and all other apps on the system.

In such cases, large amounts will then be frequently swapped out to the OS swap file. There is only so much physical memory available. Your trivial comment to "buy more RAM" in such cases is silly. Nobody wants to buy more RAM because one process is a memory hog. That silly comment applies just as well to using RAMDirectory. If RAM is unlimited then the memory map is pointless. The reason is that we don't want to buy more RAM whenever a process is a memory hog.

We are seeing a lucene app with 50 gb of virtual memory. We're actually hitting the limits on the swap file, which means we are swapping out way too much memory because of the lucene memory mapping. It's true that using a different lucene directory mapping would be worse. But it's also true that the MMapDirectory is no panacea, and it has its limits. Virtual address space is not "for free". Unlimited use of virtual memory is not for free. If you are using Linux, the keyword is "swappiness".

Turning this way down should nullify everything you are ranting about. No, that's another simplistic answer. Swappiness does not solve the problem. Regardless of how much you tell the operating system to avoid swap, when a memory mapped file is requested, the operating system has no choice but to load it into physical RAM. Swappiness refers to situations of choice when the operating system may choose to cache into swap RAM that has remained unused for a while.

It has no impact when your apache lucene is a memory hog. Hi, I have a issue here. My server had a crash ans now the solr is not getting started. It is giving this error : org. NativeFSLockFactory 72bae Casically it has mapped the index directory in the viryual memory and accesing it from there. How can i delete this virtual memory casche so that it does not look at the file path mentioned. Hi rocky, this has nothing to do with virtual memory.

You index was just corrupted by the crash. The virtual memory is freed asap when the process dies. Hi Uwe, really interesting blog! But I have one thing to ask you Do you know any detail? What are the differences between the MMap approach and this new one?

I suppose the new directory implementation is better? The reason is FSDirectory. In gerenal this should be configureable, means NRTCachingDirectory should be configureable to take another DirectoryFactory as delegate.

The idea here is to improve the NRT case, where you don't always commit your indexed data but reopen the IndexReader quite often to see changes asap. Once you commit, the data is written to disk. This dir just caches the small amounts of data before they are written to disk on commit. This directory is not really faster than MMap on reading a little bit it is - see above in blog post, because the Java overhead ByteBuffer vs. Am I Correct? Thank you in advance. Of course if the buffer is too small, it will write to disk before the hard commit.

But a commit still does not happen. The whole idea is just to delay writes as long as possible. Hi Uwe, I was going in deep more, because I want to have Clear at most the architecture : I'm going to try to be as clear as possible, because in my opinion in my previous posts I made a mistake : Near Real Time Scenario First of all we have the Ram Buffer. The Ram Buffer will ingest documents until it is full or an hard commit happens.

In that case the Ram Buffer is freed. Active Oldest Votes. You should either: Use the same RAMDirectory in both places, or Open your index in an FSDirectory , with which it will be saved to the file system, and can be reopened.

See FSDirectory. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password.

Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Making Agile work for data science. Stack Gives Back Featured on Meta. New post summary designs on greatest hits now, everywhere else eventually. When answering a question please: Read the question carefully. Understand that English isn't everyone's first language so be lenient of bad spelling and grammar. If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem.

Insults are not welcome. Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question. Let's work to help developers, not make them feel stupid. Related Questions. Memory segment issues! C - structs - files - segmentation. Segmentation fault when reading the binary file kernel. Cut multiple segments of audio in naudio C. Segmentation fault below certain character length.

Segmentation fault core dumped with fread. Segmentation fault with closed hash table.