Recent studies have proposed to dynamically reshape the power demand curve of a data center (i.e., power shaving) with energy storage devices, particularly uninterruptible power supply (UPS) batteries. Power shaving can be used to limit the peak power demand in a data center, in order to reduce both the power infrastructure investment (i.e., cap-ex) and the electricity bills (i.e., op-ex). However, power shaving requires the UPS batteries to be frequently charged/discharged, which is known to compromise the battery lifetime and availability. This paper presents a detailed quantitative study that explores different options to integrate supercapacitor (SC) with batteries for cost-efficient energy storage. Compared with batteries, SC allows more charge/discharge cycles and has a higher power density, which are desirable for fast power shaving. However, SC also has undesirable characteristics (e.g., relatively high self-discharging rate and cost). Therefore, we quantitatively compare three possible energy storage options (i.e., Battery-only, SC-only, and Battery+SC) in detail, with different SC self-discharging rate assumptions. SC options (SC-only and Battery+SC) are shown to be more cost-efficient designs, saving the energy storage cost by 34 percent, on average, compared with Battery-only. For a 10 MW data center in a 10-year period, the savings can be converted to $3 M in total cost of ownership (TCO) reduction by allowing more servers to be deployed. In addition, we also propose the integration of energy storage with dynamic voltage and frequency scaling (DVFS) to cap the peak power demand (i.e., power capping). Specifically, we comparatively studyfour power capping algorithms and discuss their applicable scenarios. Finally, we introduce our proof-of-concept SC physical testbed and present preliminary hardware testing results.