OOM error via mimedetect on low-memory systems #84

New issue

Closed

opened 2023-05-19 08:58:03 +02:00 by slowbro · 0 comments

slowbro commented

2023-05-19 08:58:03 +02:00

Issue:

Uploading a ~128MB file causes an allocation of over 1GB of memory

Log:

0x1a  |   File "/python-docker/fhost.py", line 513, in fhost
0x1a  |     return store_file(
0x1a  |            ^^^^^^^^^^^
0x1a  |   File "/python-docker/fhost.py", line 379, in store_file
0x1a  |     sf, isnew = File.store(f, requested_expiration, addr, ua, secret)
0x1a  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0x1a  |   File "/python-docker/fhost.py", line 281, in store
0x1a  |     mime = get_mime()
0x1a  |            ^^^^^^^^^^
0x1a  |   File "/python-docker/fhost.py", line 223, in get_mime
0x1a  |     guess = mimedetect.from_buffer(data)
0x1a  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0x1a  |   File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 108, in from_buffer
0x1a  |     return self._handle509Bug(e)
0x1a  |            ^^^^^^^^^^^^^^^^^^^^^
0x1a  |   File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 135, in _handle509Bug
0x1a  |     raise e
0x1a  |   File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 106, in from_buffer
0x1a  |     return maybe_decode(magic_buffer(self.cookie, buf))
0x1a  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0x1a  |   File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 307, in magic_buffer
0x1a  |     return _magic_buffer(cookie, buf, len(buf))
0x1a  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0x1a  |   File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 217, in errorcheck_null
0x1a  |     raise MagicException(err)
0x1a  | magic.MagicException: b'cannot allocate 1019473104 bytes (Cannot allocate memory)'

Steps to replicate

On a low-memory system (1GB ram)
Setup 0x0
All defaults, no vscan, no nsfw.
Upload a 127MB file
OOM Error happens

Secondary Issue

It will also not clear the uploaded file from memory after raising the error for a period of time.
So if you had enough bandwidth, you could continuously upload, cause the host computer to OOM.

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        3589  0.0 17.4 277636 172576 ?       Ss   04:37   0:06 /usr/local/bin/python /usr/local/bin/flask run -h 0.0.0.0
<uploaded a 2nd time, ctrl+r the error page>
root        3589  0.1 29.7 402084 294364 ?       Ssl  04:37   0:09 /usr/local/bin/python /usr/local/bin/flask run -h 0.0.0.0

Environment

I run 0x0 via docker using: https://gitea.slowb.ro/slowb.ro/0x1a/src/branch/master/Dockerfile which uses flask run -h 0.0.0.0

### Issue: Uploading a ~128MB file causes an allocation of over 1GB of memory ### Log: ``` 0x1a | File "/python-docker/fhost.py", line 513, in fhost 0x1a | return store_file( 0x1a | ^^^^^^^^^^^ 0x1a | File "/python-docker/fhost.py", line 379, in store_file 0x1a | sf, isnew = File.store(f, requested_expiration, addr, ua, secret) 0x1a | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 0x1a | File "/python-docker/fhost.py", line 281, in store 0x1a | mime = get_mime() 0x1a | ^^^^^^^^^^ 0x1a | File "/python-docker/fhost.py", line 223, in get_mime 0x1a | guess = mimedetect.from_buffer(data) 0x1a | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 0x1a | File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 108, in from_buffer 0x1a | return self._handle509Bug(e) 0x1a | ^^^^^^^^^^^^^^^^^^^^^ 0x1a | File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 135, in _handle509Bug 0x1a | raise e 0x1a | File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 106, in from_buffer 0x1a | return maybe_decode(magic_buffer(self.cookie, buf)) 0x1a | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 0x1a | File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 307, in magic_buffer 0x1a | return _magic_buffer(cookie, buf, len(buf)) 0x1a | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 0x1a | File "/usr/local/lib/python3.11/site-packages/magic/__init__.py", line 217, in errorcheck_null 0x1a | raise MagicException(err) 0x1a | magic.MagicException: b'cannot allocate 1019473104 bytes (Cannot allocate memory)' ``` ### Steps to replicate - On a low-memory system (1GB ram) - Setup 0x0 - All defaults, no vscan, no nsfw. - Upload a 127MB file - OOM Error happens #### Secondary Issue It will also not clear the uploaded file from memory after raising the error for a period of time. So if you had enough bandwidth, you could continuously upload, cause the host computer to OOM. ``` USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 3589 0.0 17.4 277636 172576 ? Ss 04:37 0:06 /usr/local/bin/python /usr/local/bin/flask run -h 0.0.0.0 <uploaded a 2nd time, ctrl+r the error page> root 3589 0.1 29.7 402084 294364 ? Ssl 04:37 0:09 /usr/local/bin/python /usr/local/bin/flask run -h 0.0.0.0 ``` ### Environment I run 0x0 via docker using: https://gitea.slowb.ro/slowb.ro/0x1a/src/branch/master/Dockerfile which uses `flask run -h 0.0.0.0`