[Python-il] Apache with mod WSGI (for django) crashes when you "import nltk"

Avishalom Shalit avishalom at gmail.com
Mon Oct 14 20:59:34 IST 2013


well, it was solved by someone else, and it was a while ago,
for our needs we run it not under apache, but under > python manage.py
runserver 0:8000
and it works fine.

i think at some point we were able to get it running under apache after
updating our linux distribution and apache,
but we fell back to the django dev server

-- vish



On 14 October 2013 14:26, <gritchie at gmail.com> wrote:

> Hi Vish - Did you manage to get this working somehow? I am also having
> problems using the NLTK in a django app. I'd appreciate any tips you have
> on setting things up correctly.
>
> Thanks,
>
> Graham
>
>
> On Friday, February 1, 2013 10:49:00 AM UTC, Avishalom Shalit wrote:
>
>> thanks.
>> actually this is an internal app, only available on our VPN,
>> so security is not an issue ,
>> and i only expect a maximum of 4 users
>>
>> i will look at the other setups.
>> thanks
>>
>> -- vish
>>
>>
>>
>> On 31 January 2013 23:48, Emanuel Ilyayev <emi... at gmail.com> wrote:
>>
>>>  I don't know enough NLTK but I work with django :)
>>>
>>> From Asaf's description it looks like you have to change your
>>> architecture. Apache - in it's default configuration - is not efficient in
>>> working with heavy processes because it creates a new process for each
>>> request. There are better setups like using gUnicorn or uWSGI that load n
>>> workers and distribute the work between them (usually n = number of cores X
>>> 2 + 1).
>>>
>>> More robust and scalable setup would include a separate workers that
>>> answer to the NLTK requests asynchronously and django approaches these
>>> workers via a message queue. This setup will allow you to put your NLTK
>>> workers even on a separate machine without creating situation where your
>>> web server is competing with your NLTK workers on limited resources (CPU
>>> and RAM).
>>>
>>> Even if you will eventually find the way to configure apache to load
>>> NLTK without crashing - the URL that handles NLTK requests would be a
>>> perfect point to attack you server and to bring it into a DOS (denial of
>>> service) situation using only a couple of strong machines approaching this
>>> URL....
>>>
>>> I urge you to read a little bit about gEvent and Celery to understand
>>> what I'm talking about.
>>>
>>> HTH
>>>
>>> --
>>> Emanuel
>>>
>>>
>>>
>>>
>>> On Thu, Jan 31, 2013 at 7:30 PM, asaf greenberg <asafgr... at gmail.com>wrote:
>>>
>>>>
>>>> i don't know enough django, but i worked with nltk.
>>>> NLTK is a very heavy module, lagging on import is expected, especially
>>>> if you're using certain modules.
>>>>
>>>> AFAIK you should `import' it only once, on server (re)start, and it
>>>> costs about 10-30 secs (did you optimize with *pyc or *pyo?). unless you're
>>>> short on RAM... but i hope that's not the case.
>>>>
>>>> NLTK has also many sub-modules, which can and should be disabled, for
>>>> performance.
>>>>
>>>> Does it hang elsewhere (apart from server startup)?
>>>> does it have a longer delay than 20-30 secs.?
>>>>
>>>>
>>>>
>>>> On 1/31/2013 6:44 PM, Avishalom Shalit wrote:
>>>>
>>>>    As title.
>>>>
>>>>  It just silently hangs.
>>>>
>>>>  as far as i found on google, other people have ran into it,
>>>>  but nobody posted a solution.
>>>>
>>>>  anybody overcame this before ?
>>>>
>>>>  thanks
>>>>
>>>>
>>>>  -- vish
>>>>
>>>>
>>>>
>>>> ______________________________**_________________
>>>> Python-il mailing listPyth... at hamakor.org.ilhttp://hamakor.org.il/cgi-bin/**mailman/listinfo/python-il <http://hamakor.org.il/cgi-bin/mailman/listinfo/python-il>
>>>>
>>>>
>>>>
>>>> ______________________________**_________________
>>>> Python-il mailing list
>>>> Pyth... at hamakor.org.il
>>>> http://hamakor.org.il/cgi-bin/**mailman/listinfo/python-il<http://hamakor.org.il/cgi-bin/mailman/listinfo/python-il>
>>>>
>>>>
>>>
>>> ______________________________**_________________
>>> Python-il mailing list
>>> Pyth... at hamakor.org.il
>>> http://hamakor.org.il/cgi-bin/**mailman/listinfo/python-il<http://hamakor.org.il/cgi-bin/mailman/listinfo/python-il>
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://hamakor.org.il/pipermail/python-il/attachments/20131014/33a9a8f2/attachment.html>


More information about the Python-il mailing list