Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle panic during validating admission webhook admission #108746

Merged
merged 1 commit into from May 19, 2022
Merged

Handle panic during validating admission webhook admission #108746

merged 1 commit into from May 19, 2022

Conversation

deads2k
Copy link
Contributor

@deads2k deads2k commented Mar 16, 2022 •
edited

Validating admission webhook evaluation can fail, if uncaught this
crashes a kube-apiserver. Add handling to catch panic while preserving
the behavior of "must not fail".

/kind bug
/priority important-soon
/sig api-machinery

Panics while calling validating admission webhook are caught and honor the fail open or fail closed setting.

@deads2k deads2k mentioned this pull request Mar 16, 2022
Closed
@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 16, 2022
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 16, 2022
@leilajal
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 17, 2022
@deads2k deads2k force-pushed the proof-2 branch 2 times, most recently from 2bd12bf to ba53c2d Compare May 12, 2022 19:41
@deads2k deads2k force-pushed the proof-2 branch 3 times, most recently from 726ecba to 13a35f1 Compare May 17, 2022 13:51
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 17, 2022
@deads2k deads2k changed the title proof of panic handling Handle panic during validating admission webhook admission May 17, 2022
@deads2k
Copy link
Contributor Author

deads2k commented May 17, 2022

/hold cancel
/kind bug

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels May 17, 2022
@deads2k
Copy link
Contributor Author

deads2k commented May 17, 2022

/priority important-soon

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 17, 2022
liggitt
liggitt reviewed May 17, 2022
View reviewed changes
staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go Outdated
defer wg.Done()
defer func() {
// HandleCrash has already called the crash handlers and it has been configured to utilruntime.ReallyCrash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: weird double spaces in comments here and above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: weird double spaces in comments here and above

I miss my double spaces after periods

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/testing/testcase.go
}},
ExpectStatusCode: http.StatusInternalServerError,
ExpectAllow: false,
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no panic-related error message here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no panic-related error message here?

added

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go Outdated
}
if ignoreClientCallFailures {
// if failures are supposed to ignored, ignore it
klog.Warningf("Panic calling webhook, failing open %v: %v", invocation.Webhook.GetName(), r)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't invocation.Webhook.GetName() the same as hookName? did you mean GetConfigurationName()?

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go Outdated
return
}
// this ensures that the admission request fails and a message is provided.
errCh <- apierrors.NewInternalError(fmt.Errorf("ValidatingAdmissionWebhook/%v has panicked: %v", invocation.Webhook.GetName(), r))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't invocation.Webhook.GetName() the same as hookName? did you mean GetConfigurationName()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't invocation.Webhook.GetName() the same as hookName? did you mean GetConfigurationName()?

other errors here use hookName. Updated for variable instead of method.

@liggitt
Copy link
Member

liggitt commented May 18, 2022

mm... looks like the unit test needs to tolerate whichever message comes first:

plugin_test.go:332: match & fail (but fail because fail closed): expected an error saying "ValidatingAdmissionWebhook/internalErr B has panicked: Start panicking!", but got Internal error occurred: ValidatingAdmissionWebhook/internalErr C has panicked: Start panicking!

@deads2k
Handle panic during validating admission webhook admission
d412bf9 
Validating admission webhook evaluation can fail, if uncaught this
crashes a kube-apiserver.  Add handling to catch panic while preserving
the behavior of "must not fail".
@liggitt
Copy link
Member

liggitt commented May 19, 2022

/retest

@liggitt
Copy link
Member

liggitt commented May 19, 2022

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 19, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit b215a89 into kubernetes:master May 19, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.25 milestone May 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liggitt liggitt liggitt left review comments

@aojea aojea aojea left review comments

@cheftako cheftako Awaiting requested review from cheftako

@enj enj Awaiting requested review from enj

Assignees

@liggitt liggitt

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.25
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@deads2k @leilajal @liggitt @k8s-ci-robot @aojea