Runtime Error in Paid Space after upgrade and no info
Hi,We have this space with a gradio app that was running perfectly last week with the free plan:huggingface.coOlas Prediction Live Dashboard - a Hugging Face Space by valoryDiscover amazing ML apps made by the communityAnd since monday this week we got a Runtime Error saying:# Runtime error
## Memory limit exceeded (16Gi)The build logs were fine and the errors comes with the start up of the container. We decided to go for a paid version and after updating the error we have now is:# Runtime error
## Memory limit exceeded (32Gi)Please someone from infra, could you give us more information about what is going on here and how to solve this? We need to present a poster in a conference next week where we want to share this dashboard.@radames@michellehbnMany thanksRosaP.S.: I checked the forum and it was not helpful any similar posts since we have no info about the problem.
2024-05-24T15:10:04Z
hi@cyberosa, could you trace your app memory footprint? perhaps running it locally to see if it’s not growing about the Space hardware limit?
2024-05-24T15:11:42Z
Hi@radames,The app is working perfectly fine locally so we cannot reproduce the issue. Do you have any other logs of the container that could give us info why the app is working locally but not working in HF?BestP.S.: My local machine is a Mac-Pro M3 with 18 Gb RAM and I monitored the execution and could not detect any error.
2024-05-24T15:16:08Z
Trying to restart the space again…
2024-05-24T15:18:42Z
Ok it did not workCould someone increase the verbose of the docker logs so that I can get more information?
2024-05-24T15:19:59Z
I recommend you logging the memory on your app and see the memory on logs both locally and on Spaces.In terms of logs on Spaces, the log tab is all we could access as well.Let me try running it here locally, I’ll let you know
2024-05-24T15:23:00Z
It would be helpful if someone could kill the container in HF, delete the old image to force building a new one. In my own experience with docker, and other colleagues confirm the same, sometimes rebuilding but without removing the old image does not work.Many thanks for your help. Looking forward to your own local check
2024-05-24T15:25:01Z
cyberosa:It would be helpful if someone could kill the container in HF, delete the old image to force building a new one. In my own experience with docker, and other colleagues confirm the same, sometimes rebuilding but without removing the old image does not work.You can do that by doing a factory reboot. Have you tried it?
2024-05-24T15:26:29Z
Factory rebuildClick this button to trigger a factory rebuild of your Space. This will invalidate Docker layer caches and rebuild your space from scratch, reinstalling all dependencies.Was not aware of this option… let me try
2024-05-24T15:36:08Z
No. It did not help. Same error. This is the memory consumption of the app running on my computer:Screenshot 2024-05-24 at 17.35.491064×164 21.4 KBI am playing with the app and it works perfectly fine.  Any other ideas?
2024-05-24T17:13:42Z
hi it seems like there is a memory spike onprepare_data()which is causing the OOM issue.
2024-05-25T15:20:47Z
Many thanks@radamesMay I ask which tool did you use to identify the source of trouble?I will work on refactoring the app and try again my luck next week
2024-05-28T18:39:25Z
on linux you can runhtopand watch memory usage, it peaked over 28.7RAMimage1258×136 17.9 KB
2024-05-29T07:26:08Z
In the end, the issue was in the way we were computing errors in one parquet file. The app was parsing the data in the wrong way. Once that was changed the app is running again. Anyway I detected some performance improvements that can be done and we will be working on that soon.Many thanks for your helpful support.
2024-05-29T18:30:42Z
Amazing! I’m glad it worked!
2024-05-30T06:30:58Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.