/bin/bash: Exec format error

macphp asked:

I have a weird issue on an Ubuntu 10.04 LTS virtual server. A few days ago I started getting these alerts:

***** Nagios *****

Notification Type: PROBLEM

Service: Current Load
Host: localhost
Address: 127.0.0.1
State: CRITICAL

Date/Time: Mon Aug 13 13:09:58 EDT 2012

Now logging in to the server does not work. It seems like I am able to authenticate with ssh but then seems to fail at getting a prompt. I get:

/bin/bash: Exec format error
Connection to 192.172.132.20 closed.

I can’t login from any account. Can’t try single user as it is remotely hosted. I can connect without starting a shell using ssh -N.

I’ve received this from cron. It seems like all my /bin files are gone. Is there any way to recover?

/etc/cron.daily/apt:
/bin/sleep: 2: /bin/sleep: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/sleep: 1: /bin/sleep: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/sleep: 2: /bin/sleep: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep:  ELF    : not found
/bin/sleep: 1: /bin/sleep: Syntax error: word unexpected (expecting ")")
/bin/pidof: 2: /bin/pidof: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/pidof: 1: /bin/pidof: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/pidof: 2: /bin/pidof: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof:  ELF    : not found
/bin/pidof: 1: /bin/pidof: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/dpkg:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp:  ELF    : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/standard:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp:  ELF    : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/bin/awk: 2: /bin/awk: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/awk: 1: /bin/awk: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/awk: 2: /bin/awk: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/awk: 1: /bin/awk: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/awk: 1: /bin/awk:  ELF    : not found
/bin/awk: 1: /bin/awk: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/thedailycommandline:
run-parts: failed to exec /etc/cron.daily/thedailycommandline: Exec format error
run-parts: /etc/cron.daily/thedailycommandline exited with return code 1

Any help would be greatly appreciated.

My answer:


That paste looks like severe filesystem corruption.

At this point, were it my server, I would:

  • Check the disks for hardware failure and replace any disk that has failed, is failing, or even looks like it might fail in the near future.
  • Reinstall the OS and restore my data from the most recent good backup, checking them carefully to ensure that I’m not restoring corrupt data. This could have been going on for quite a long time without your knowledge, so check your backup carefully before restoring it.

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.