2025-踩坑记录

发表于 分类于 阅读次数：

环境问题 - 2025-12-10

edu-*-cmall-web 新增 xxl-job，beta环境没有问题，上生产服务就起不来

1
2
3
4
5
6
7
8
9
10
{"date":"2025-12-10 11:45:29.405","level":"INFO","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.s.s.c.ThreadPoolTaskScheduler Shutting down ExecutorService 'taskScheduler' "}
{"date":"2025-12-10 11:45:29.417","level":"INFO","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.s.s.c.ThreadPoolTaskExecutor Shutting down ExecutorService 'applicationTaskExecutor' "}
{"date":"2025-12-10 11:45:29.423","level":"ERROR","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"c.x.j.core.executor.XxlJobExecutor null java.lang.NullPointerException: null\tat com.xxl.job.core.executor.XxlJobExecutor.stopEmbedServer(XxlJobExecutor.java:159)\tat com.xxl.job.core.executor.XxlJobExecutor.destroy(XxlJobExecutor.java:85)\tat com.xxl.job.core.executor.impl.XxlJobSpringExecutor.destroy(XxlJobSpringExecutor.java:55)\tat org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:258)\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587)\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559)\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092)\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520)\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085)\tat org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061)\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:405)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\tat com.*.cmall.Application.main(Application.java:39)\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\tat java.lang.reflect.Method.invoke(Method.java:498)\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:107)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) "}
{"date":"2025-12-10 11:45:29.661","level":"INFO","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.a.catalina.core.StandardService Stopping service [Tomcat] "}
{"date":"2025-12-10 11:45:29.688","level":"INFO","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.s.b.a.l.ConditionEvaluationReportLoggingListener Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. "}
{"date":"2025-12-10 11:45:29.691","level":"WARN","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"c.h.tech.core.util.SpringUtils get beanClass:interface org.springframework.cloud.kubernetes.PodUtils occurs error:{} java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@69997e9d has not been refreshed yet\tat org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1096)\tat org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1126)\tat com.*.tech.core.util.SpringUtils.getBean(SpringUtils.java:27)\tat com.*.tech.core.util.K8sUtils.getPodUtils(K8sUtils.java:140)\tat com.*.tech.core.util.K8sUtils.getSelfIp(K8sUtils.java:69)\tat org.seeplnframework.skyeye.common.util.LogUtils.obtainLogData(LogUtils.java:57)\tat org.seeplnframework.skyeye.logback.api.filter.LogbackFilter.decide(LogbackFilter.java:59)\tat org.seeplnframework.skyeye.logback.api.filter.LogbackFilter.decide(LogbackFilter.java:27)\tat ch.qos.logback.core.spi.FilterAttachableImpl.getFilterChainDecision(FilterAttachableImpl.java:57)\tat ch.qos.logback.core.UnsynchronizedAppenderBase.getFilterChainDecision(UnsynchronizedAppenderBase.java:133)\tat ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:79)\tat ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)\tat ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)\tat ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)\tat ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)\tat ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)\tat ch.qos.logback.classic.Logger.log(Logger.java:765)\tat org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:216)\tat org.springframework.boot.SpringApplication.reportFailure(SpringApplication.java:837)\tat org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:812)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:325)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\tat com.*.cmall.Application.main(Application.java:39)\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\tat java.lang.reflect.Method.invoke(Method.java:498)\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:107)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) "}
{"date":"2025-12-10 11:45:29.691","level":"WARN","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.s.s.l.api.filter.LogbackFilter log skyeye >>> LogbackFilter occur exception java.lang.IllegalStateException: podUitls not found\tat com.*.tech.core.util.K8sUtils.getPodUtils(K8sUtils.java:142)\tat com.*.tech.core.util.K8sUtils.getSelfIp(K8sUtils.java:69)\tat org.seeplnframework.skyeye.common.util.LogUtils.obtainLogData(LogUtils.java:57)\tat org.seeplnframework.skyeye.logback.api.filter.LogbackFilter.decide(LogbackFilter.java:59)\tat org.seeplnframework.skyeye.logback.api.filter.LogbackFilter.decide(LogbackFilter.java:27)\tat ch.qos.logback.core.spi.FilterAttachableImpl.getFilterChainDecision(FilterAttachableImpl.java:57)\tat ch.qos.logback.core.UnsynchronizedAppenderBase.getFilterChainDecision(UnsynchronizedAppenderBase.java:133)\tat ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:79)\tat ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)\tat ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)\tat ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)\tat ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)\tat ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)\tat ch.qos.logback.classic.Logger.log(Logger.java:765)\tat org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:216)\tat org.springframework.boot.SpringApplication.reportFailure(SpringApplication.java:837)\tat org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:812)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:325)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\tat com.*.cmall.Application.main(Application.java:39)\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\tat java.lang.reflect.Method.invoke(Method.java:498)\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:107)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) "}
{"date":"2025-12-10 11:45:29.690","level":"ERROR","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.s.boot.SpringApplication Application run failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name '*.*.homework.facade.api.HomeworkInternalServiceFacade': Unexpected exception during bean creation; nested exception is java.lang.IllegalStateException: No fallback instance of type class *.*.homework.facade.fallback.HomeworkInternalServiceFacadeFallback found for feign client homeworInternalServiceFacade\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:529)\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)\tat org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1109)\tat com.xxl.job.core.executor.impl.XxlJobSpringExecutor.initJobHandlerMethodRepository(XxlJobSpringExecutor.java:88)\tat com.xxl.job.core.executor.impl.XxlJobSpringExecutor.afterSingletonsInstantiated(XxlJobSpringExecutor.java:39)\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:914)\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:405)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\tat com.*.cmall.Application.main(Application.java:39)\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\tat java.lang.reflect.Method.invoke(Method.java:498)\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:107)\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)Caused by: java.lang.IllegalStateException: No fallback instance of type class *.*.homework.facade.fallback.HomeworkInternalServiceFacadeFallback found for feign client homeworInternalServiceFacade\tat com.*.cloud.sentinel.feign.SentinelFeign$Builder$1.getFromContext(SentinelFeign.java:131)\tat com.*.cloud.sentinel.feign.SentinelFeign$Builder$1.create(SentinelFeign.java:110)\tat feign.ReflectiveFeign.newInstance(ReflectiveFeign.java:64)\tat feign.Feign$Builder.target(Feign.java:269)\tat org.springframework.cloud.openfeign.HystrixTargeter.target(HystrixTargeter.java:38)\tat org.springframework.cloud.openfeign.FeignClientFactoryBean.loadBalance(FeignClientFactoryBean.java:352)\tat org.springframework.cloud.openfeign.FeignClientFactoryBean.getTarget(FeignClientFactoryBean.java:390)\tat org.springframework.cloud.openfeign.FeignClientFactoryBean.getObject(FeignClientFactoryBean.java:361)\tat org.springframework.cloud.openfeign.FeignClientsRegistrar.lambda$registerFeignClient$0(FeignClientsRegistrar.java:246)\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.obtainFromSupplier(AbstractAutowireCapableBeanFactory.java:1230)\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1172)\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)\t... 26 common frames omitted "}
{"date":"2025-12-10 11:45:29.793","level":"INFO","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.s.s.c.a.PushAlarmMessageTask log skyeye >>> send wechat key: a9322063-76f3-4a66-ae21-46af6fb64f05, timestamp: 1765338329793 "}
{"date":"2025-12-10 11:45:30.291","level":"INFO","traceId":"","type":"BUSINESS","userId":"null","url":"","msg":"o.s.s.c.a.PushAlarmMessageTask log skyeye >>> send wechat result: {\"StatusCode\":0,\"StatusMessage\":\"success\",\"code\":0,\"data\":{},\"msg\":\"success\"} "}
  • 本次升级了kafka，增加了traceId
  • 新增了xxl-job

先把kafka降级，服务还是起不来

去掉xxl-job，服务起来了

原因是

xxl-job 在初始化时会扫描所有 Bean 来注册 JobHandler：

XxlJobSpringExecutor.initJobHandlerMethodRepository()

扫描过程中触发了 HomeworkInternalServiceFacade 这个 Feign 客户端的创建，然后发现缺少 fallback 类就报错了。

本质原因是 Feign 熔断开关 两个环境配置不一样

原因是：

解决方法：

时间反序列化问题 - 2025-06-19

海外支付，接入onerway支付，支付回调时，时间反序列化有问题，回调给的是北京时间，字符串

直接反序列化时时间比北京时间快了8个小时。

https://docs.onerway.com/apis/zh/v0.6/transaction-notification

代码中接受直接将字符串转成Date，没有加时区，造成时间有问题。

1
2
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
private Date txnTime;

解决：加上时区，这样就是北京时间

1
2
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
private Date txnTime;

还有另一个解决方式，接受使用String 接收，然后手动转成Date，这样肯定不会有问题。线上解决使用第一种方式，这样方式最快。

配置问题 - 2025-02-26

业务接入飞书审批时，明明在apollo配置了审批code，确报错，

看了代码，apollo没有配置才会报这个错，明明配置了。

查看日志，先是在腾讯云查看日志，没有搜到对应日志。进入pod查看也没有，也是奇怪了。还怀疑日志收集有问题。

最后结果是业务配置错了，直接请求到dev去了。

k8s权限问题 - 2025-02-24

java 服务 在k8s 新建的namespace 起不来，一直卡住.

但在原来的namespace 能起来。k8s新建了一个集群，原因是服务调用k8s注册发现调不通。

如何定位到的：

容器中执行健康检查报如下错误：

curl http://localhost:9858/actuator/health

应该是新版k8s把这个权限回收了。devops手动修改解决

1
{"status":"DOWN","components":{"db":{"status":"UP","details":{"database":"MySQL","validationQuery":"isValid()"}},"discoveryComposite":{"status":"DOWN","components":{"discoveryClient":{"status":"DOWN","details":{"error":"io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.111.0.1/api/v1/namespaces/shield/services. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. services is forbidden: User \"system:serviceaccount:shield:default\" cannot list resource \"services\" in API group \"\" in the namespace \"shield\"."}}}},"diskSpace":{"status":"UP","details":{"total":126423846912,"free":107297361920,"threshold":10485760,"exists":true}},"kubernetes":{"status":"UP","details":{"inside":false}},"livenessState":{"status":"UP"},"ping":{"status":"UP"},"readinessState":{"status":"UP"},"redis":{"status":"UP","details":{"version":"7.0.15"}},"refreshScope":{"status":"UP"}},"groups":["liveness","readiness"]}/

maven 问题- 2025-01-09

宙斯发布client，maven发布jar不成功，报错如下

https://zeus.corp.*.club/portal/app/reviewPublishTask?appCode=business-*-cos-client&jenkinsBuildId=10&env=prod&appType=JAR

1
2
3
4
5
6
7
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project *-cos-client: Failed to deploy artifacts: Could not transfer artifact com.*:*-cos-client:jar:20241211 from/to *-deploy-release (http://nexus3.corp.*.club/repository/maven-releases/): Failed to transfer file: http://nexus3.corp.*.club/repository/maven-releases/com/*/*-cos-client/20241211/*-cos-client-20241211.jar. Return code is: 400, ReasonPhrase: Repository does not allow updating assets: maven-releases. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

刚开始以为是宙斯maven有问题，最后看详细错误才发现，代码合错分支了，合到之前的迭代分支，之前版本号已经发过maven-releases了，再次发相同的版本号就会报这个错误 Return code is: 400, ReasonPhrase: Repository does not allow updating assets: maven-releases.

重新合到正确的迭代分支，发布成功。还有一个问题服务发布要选择正确的k8s namespace。