Skip to content

Can't upgrade w/ grafana.persistence.enabled=true #188

Description

@vdice

On GKE, upgrading with grafana persistence enabled leads to a Failed to attach volume error:

 $ k version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:53:09Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

 $ cat values-only-grafana-persistent.yaml
monitor:
  grafana:
    persistence:
      enabled: true
  influxdb:
    persistence:
      enabled: false

 $ helm install workflow-dev/workflow --namespace deis --name deis-workflow -f values-only-grafana-persistent.yaml
...
 $ # (wait until all up and running)

 $ helm upgrade deis-workflow -f values-only-grafana-persistent.yaml workflow/workflow
...
 $ kd get po | grep grafana
deis-monitor-grafana-1674584155-qz1pq    0/1       ContainerCreating   0          40s
deis-monitor-grafana-2030706665-cg04s    1/1       Running             0          4m

$ kd describe po deis-monitor-grafana-1674584155-qz1pq
Name:		deis-monitor-grafana-1674584155-qz1pq
Namespace:	deis
Node:		gke-vrd-default-pool-7890514e-p65q/10.240.0.22
Start Time:	Tue, 28 Mar 2017 12:10:58 -0600
Labels:		app=deis-monitor-grafana
		pod-template-hash=1674584155
Status:		Pending
IP:
Controllers:	ReplicaSet/deis-monitor-grafana-1674584155
Containers:
  deis-monitor-grafana:
    Container ID:
    Image:		quay.io/deis/grafana:v2.8.0
    Image ID:
    Port:		3500/TCP
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /var/lib/grafana from grafana-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-f1372 (ro)
    Environment Variables:
      INFLUXDB_URLS:		http://$(DEIS_MONITOR_INFLUXAPI_SERVICE_HOST):$(DEIS_MONITOR_INFLUXAPI_SERVICE_PORT_TRANSPORT)
      BIND_PORT:		3500
      DEFAULT_USER:		admin
      DEFAULT_USER_PASSWORD:	admin
Conditions:
  Type		Status
  Initialized 	True
  Ready 	False
  PodScheduled 	True
Volumes:
  grafana-data:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:	deis-monitor-grafana
    ReadOnly:	false
  default-token-f1372:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-f1372
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  1m		1m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned deis-monitor-grafana-1674584155-qz1pq to gke-vrd-default-pool-7890514e-p65q
  1m		14s		7	{controller-manager }			Warning		FailedMount	Failed to attach volume "pvc-5d2452ce-13e1-11e7-b75d-42010a80013f" on node "gke-vrd-default-pool-7890514e-p65q" with: googleapi: Error 400: The disk resource 'projects/deis-sandbox/zones/us-central1-b/disks/gke-vrd-c820edac-dynam-pvc-5d2452ce-13e1-11e7-b75d-42010a80013f' is already being used by 'projects/deis-sandbox/zones/us-central1-b/instances/gke-vrd-default-pool-7890514e-0xxs'

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions