links
371 shaares
Curl passing two Query string or arguments:
The -G argument will append the data specified in -d and --data-urlencode arguments at the end of the request URL, joining all data pieces with the & character and separating them from the URL with the ? character.
Example:
curl -vv -G -d "track=randwick" -d "race=1" https://s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com/puntwit
> GET /puntwit?track=randwick&race=1 HTTP/2
> Host: s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Tue, 08 Jun 2021 03:18:54 GMT
< content-type: application/json
< content-length: 1017
< apigw-requestid: AlgwyjqBywMEMSw=
<
* Connection #0 to host s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com left intact
{"version":"2.0","routeKey":"GET /puntwit","rawPath":"/puntwit","rawQueryString":"track=randwick&race=1","headers":{"accept":"*/*","content-length":"0","host":"s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com","user-agent":"curl/7.58.0","x-amzn-trace-id":"Root=1-60b9d48e-3c1555b220ef70ce1f7d2933","x-forwarded-for":"13.54.150.130","x-forwarded-port":"443","x-forwarded-proto":"https"},"queryStringParameters":{"race":"1","track":"randwick"},"requestContext":{"accountId":"302766791300","apiId":"s9iwqktpp8","domainName":"s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com","domainPrefix":"s9iwqktpp8","http":{"method":"GET","path":"/puntwit","protocol":"HTTP/1.1","sourceIp":"13.54.150.130","userAgent":"curl/7.58.0"},"requestId":"AY4mSj-ASwMENlA=","routeKey":"GET /puntwit","stage":"$default","time":"04/Jun/2021:07:21:50 +0000","timeEpoch":1622791310453},"isBase64Encoded":false}
The -G argument will append the data specified in -d and --data-urlencode arguments at the end of the request URL, joining all data pieces with the & character and separating them from the URL with the ? character.
Example:
curl -vv -G -d "track=randwick" -d "race=1" https://s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com/puntwit
> GET /puntwit?track=randwick&race=1 HTTP/2
> Host: s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Tue, 08 Jun 2021 03:18:54 GMT
< content-type: application/json
< content-length: 1017
< apigw-requestid: AlgwyjqBywMEMSw=
<
* Connection #0 to host s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com left intact
{"version":"2.0","routeKey":"GET /puntwit","rawPath":"/puntwit","rawQueryString":"track=randwick&race=1","headers":{"accept":"*/*","content-length":"0","host":"s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com","user-agent":"curl/7.58.0","x-amzn-trace-id":"Root=1-60b9d48e-3c1555b220ef70ce1f7d2933","x-forwarded-for":"13.54.150.130","x-forwarded-port":"443","x-forwarded-proto":"https"},"queryStringParameters":{"race":"1","track":"randwick"},"requestContext":{"accountId":"302766791300","apiId":"s9iwqktpp8","domainName":"s9iwqktpp8.execute-api.ap-southeast-2.amazonaws.com","domainPrefix":"s9iwqktpp8","http":{"method":"GET","path":"/puntwit","protocol":"HTTP/1.1","sourceIp":"13.54.150.130","userAgent":"curl/7.58.0"},"requestId":"AY4mSj-ASwMENlA=","routeKey":"GET /puntwit","stage":"$default","time":"04/Jun/2021:07:21:50 +0000","timeEpoch":1622791310453},"isBase64Encoded":false}
Good breakdown of printing timing breakdown of curl. For example command:
curl -L --output /dev/null --silent --show-error --write-out 'lookup: %{time_namelookup}\nconnect: %{time_connect}\nappconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nredirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n' 'https://google.com'
lookup: 0.038
connect: 0.063
appconnect: 0.177
pretransfer: 0.178
redirect: 0.225
starttransfer: 0.679
total: 0.929
From the man pages -
lookup: The time, in seconds, it took from the start until the name resolving was completed.
connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
total: The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.
curl -L --output /dev/null --silent --show-error --write-out 'lookup: %{time_namelookup}\nconnect: %{time_connect}\nappconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nredirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n' 'https://google.com'
lookup: 0.038
connect: 0.063
appconnect: 0.177
pretransfer: 0.178
redirect: 0.225
starttransfer: 0.679
total: 0.929
From the man pages -
lookup: The time, in seconds, it took from the start until the name resolving was completed.
connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
total: The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.
Host Anycast Service
tcp slow start fast retransmission's congestion duplicate acknowledgements
Setting Max Segment Size (MSS) via iptables
# Add rules
$ sudo iptables -I OUTPUT -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 48
# delete rules
$ sudo iptables -D OUTPUT -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 48
# Add rules
$ sudo iptables -I OUTPUT -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 48
# delete rules
$ sudo iptables -D OUTPUT -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 48
weird MTU in wireshark - result of TCP segmentation offload
How to set manually set tcp keepalives in curl rather than using Linux Kernel default tcp keepalive - 2 hours (7200 seconds)
Linux Kernel Default Values Ref: https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html
----------------------------------------------------------------------------------------------------------------------------------------------
ubuntu@ip-10-0-135-84:~$ sudo cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
ubuntu@ip-10-0-135-84:~$ sudo cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
ubuntu@ip-10-0-135-84:~$ sudo cat /proc/sys/net/ipv4/tcp_keepalive_probes
9
--keepalive-time <seconds>
This option sets the time a connection needs to remain idle before sending keepalive probes and the time between individual
keepalive probes. It is currently effective on operating systems offering the TCP_KEEPIDLE and TCP_KEEPINTVL socket options (mean‐
ing Linux, recent AIX, HP-UX and more). This option has no effect if --no-keepalive is used.
If this option is used several times, the last one will be used. If unspecified, the option defaults to 60 seconds.
Added in 7.18.0
Linux Kernel Default Values Ref: https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html
----------------------------------------------------------------------------------------------------------------------------------------------
ubuntu@ip-10-0-135-84:~$ sudo cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
ubuntu@ip-10-0-135-84:~$ sudo cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
ubuntu@ip-10-0-135-84:~$ sudo cat /proc/sys/net/ipv4/tcp_keepalive_probes
9
--keepalive-time <seconds>
This option sets the time a connection needs to remain idle before sending keepalive probes and the time between individual
keepalive probes. It is currently effective on operating systems offering the TCP_KEEPIDLE and TCP_KEEPINTVL socket options (mean‐
ing Linux, recent AIX, HP-UX and more). This option has no effect if --no-keepalive is used.
If this option is used several times, the last one will be used. If unspecified, the option defaults to 60 seconds.
Added in 7.18.0
Chaos Engineering is the discipline of experimenting on a distributed system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.
load balancing, affinity, persistence, sticky sessions: what you need to know - HAProxy Technologies
What is the difference between Persistence and Affinity
Affinity: this is when we use an information from a layer below the application layer to maintain a client request to a single server
Persistence: this is when we use Application layer information to stick a client to a single server
sticky session: a sticky session is a session maintained by persistence
The main advantage of the persistence over affinity is that it’s much more accurate, but sometimes, Persistence is not doable, so we must rely on affinity.
Using persistence, we mean that we’re 100% sure that a user will get redirected to a single server.
Using affinity, we mean that the user may be redirected to the same server…
Affinity: this is when we use an information from a layer below the application layer to maintain a client request to a single server
Persistence: this is when we use Application layer information to stick a client to a single server
sticky session: a sticky session is a session maintained by persistence
The main advantage of the persistence over affinity is that it’s much more accurate, but sometimes, Persistence is not doable, so we must rely on affinity.
Using persistence, we mean that we’re 100% sure that a user will get redirected to a single server.
Using affinity, we mean that the user may be redirected to the same server…
50 user SSO configuration with AAD. Salesforce has ability to provision sandboxed test environments for dev/test.