关注小众语言、AI技术,记录、分享技术点滴!

0%

golang中net.http请求时header中的Content-Length无效坑

在一次和国内某云厂商对接对象存储时,因为他们对象存储服务,是直接使用的是第三方开源服务,开源服务默认并不支持chunked编码(而国内其它大厂一般都是支持chunked编码)。导致上传文件一直报HTTP/1.1 411 Length Required错误,出现问题的原因是说,我没有传Content-Length。

而明明我在header里设置了Content-Length,但是抓包发现,实际却变成了chunked

经过反复的测试情况都依旧,直到在github上找到了这个https://github.com/golang/go/issues/16264

然后查看了golang源码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
func NewRequest(method, url string, body io.Reader) (*Request, error) {
if method == "" {
// We document that "" means "GET" for Request.Method, and people have
// relied on that from NewRequest, so keep that working.
// We still enforce validMethod for non-empty methods.
method = "GET"
}
if !validMethod(method) {
return nil, fmt.Errorf("net/http: invalid method %q", method)
}
u, err := parseURL(url) // Just url.Parse (url is shadowed for godoc).
if err != nil {
return nil, err
}
rc, ok := body.(io.ReadCloser)
if !ok && body != nil {
rc = ioutil.NopCloser(body)
}
// The host's colon:port should be normalized. See Issue 14836.
u.Host = removeEmptyPort(u.Host)
req := &Request{
Method: method,
URL: u,
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(Header),
Body: rc,
Host: u.Host,
}
if body != nil {
switch v := body.(type) {
case *bytes.Buffer:
req.ContentLength = int64(v.Len())
buf := v.Bytes()
req.GetBody = func() (io.ReadCloser, error) {
r := bytes.NewReader(buf)
return ioutil.NopCloser(r), nil
}
case *bytes.Reader:
req.ContentLength = int64(v.Len())
snapshot := *v
req.GetBody = func() (io.ReadCloser, error) {
r := snapshot
return ioutil.NopCloser(&r), nil
}
case *strings.Reader:
req.ContentLength = int64(v.Len())
snapshot := *v
req.GetBody = func() (io.ReadCloser, error) {
r := snapshot
return ioutil.NopCloser(&r), nil
}
default:
// This is where we'd set it to -1 (at least
// if body != NoBody) to mean unknown, but
// that broke people during the Go 1.8 testing
// period. People depend on it being 0 I
// guess. Maybe retry later. See Issue 18117.
}
// For client requests, Request.ContentLength of 0
// means either actually 0, or unknown. The only way
// to explicitly say that the ContentLength is zero is
// to set the Body to nil. But turns out too much code
// depends on NewRequest returning a non-nil Body,
// so we use a well-known ReadCloser variable instead
// and have the http package also treat that sentinel
// variable to mean explicitly zero.
if req.GetBody != nil && req.ContentLength == 0 {
req.Body = NoBody
req.GetBody = func() (io.ReadCloser, error) { return NoBody, nil }
}
}

return req, nil
}

可以看到,这里面居然有个switch,当你使用bytes.Buffer,bytes.Reader或者strings.Reader作为Body的时候,它会自动给你设置req.ContentLength…

所以,问题不是当你Post一个ReadCloser的时候,就会变成chunked,而是你Post非这三种类型的body进来的时候都没有Content-Length,需要自己显式设置。代码如下:

1
2
req, _ := http.NewRequest(method, url, bodyReader)
req.ContentLength = req.Header.Get("Content-Length")

设置了req.ContentLength之后,抓包如下确实不再是chunked编码了,抓包如下

至此问题解决。