AWS: search-prod1 is nonresponsive and stuck in "stopping" state

Description

  • We received a notification from AWS at 6:27 PM EST on 1/7/15 about an unhealthy search node. search-prod1 became nonresponsive; attempts to connect to it via SSH, etc. failed.

  • I have tried to reboot the instance from the EC2 console panel, with no obvious success.

  • I have tried to stop the instance; it got stuck in the stopping state.

  • I tried to stop it again, which should have run a "force stop" state, but it remains stuck in the "stopping" state.

Activity

Show:
Mark Matienzo
January 8, 2015, 12:01 AM
  • The search cluster otherwise appears healthy, and all shards have a replica.

  • @search-prod1@ finally responded to the stop command, and so I asked it to start up again. It's currently in the "pending" state.

Mark Matienzo
January 8, 2015, 12:08 AM

@search-prod1@ is back online and has rejoined the cluster.

Mark Matienzo
January 8, 2015, 12:09 AM

Note: EC2 console still says that search-prod1 is @pending@.

Mark Matienzo
January 8, 2015, 12:11 AM

OK, back to @running@. Will monitor and close this later.

Done

Assignee

Mark Matienzo

Reporter

Mark Matienzo

Labels

None

Priority

Highest
Configure